Method and apparatus for distributed processing of file

ABSTRACT

A method and apparatus for the distributed processing of a file are disclosed. The apparatus includes a shared data block selection unit, a modified data block selection unit, a first file distributed-processing unit, and a second file distributed-processing unit. The shared data block selection unit selects at least one first data block to remain without change after distributed processing of a file. The modified data block selection unit selects at least one second data block to be modified after the distributed processing of a file. The first file distributed-processing unit allows an inode after the distributed processing of a file to point to the first data blocks so that the first data block is shared before and after the distributed processing of a file. The second file distributed-processing unit allows the inode after the distributed processing of a file to point to at least one third data block.

CROSS REFERENCE TO RELATED APPLICATION

This application claims the benefit of Korean Patent Application No.10-2013-0058231, filed on May 23, 2013, which is hereby incorporated byreference in its entirety into this application.

BACKGROUND OF THE INVENTION

1. Technical Field

The present invention relates generally to a method and apparatus forthe distributed processing of a file and, more particularly, to a methodand apparatus for the distributed processing of a file that are capableof efficiently performing segmentation, merging, and front addition inconnection with a large file.

2. Description of the Related Art

A conventional file system provides only file open, read, write,end-add, end-truncate, and close operations. In computer systems used ingenome and protein analyses, efficient tasks cannot be performed usingonly the operations that are provided by the conventional file system asdescribed above.

The size of an input data file for a genome analysis application is verylarge (e.g., 218 GB), and the time it takes to analyze the content ofthe data file is very long. In order to reduce analysis time, a datafile is segmented into a plurality of small files, the small files areprocessed in parallel, the processed files are merged into a singlelarge file, and the single large file is used as input in a subsequentstage.

As described above, the conventional file system offsets a dataparallelism effect because it consumes a lot of time to perform thetasks of segmenting a large file and merging small files.

Furthermore, in the conventional file system, in order to fragment afile, the large input/output bandwidths of a data storage device areused because the original file needs to be read and written intomultiple files, thereby deteriorating system efficiency and performance.

In connection with this, Korean Patent Application Publication No.10-2002-0092550 discloses a mass file storage system and a method ofdeleting and adding the data blocks of dynamic multi-level inodes usingthe system.

SUMMARY OF THE INVENTION

Accordingly, the present invention has been made keeping in mind theabove problems occurring in the conventional art, and an object of thepresent invention is to provide a method and apparatus for thedistributed processing of a file that are capable of performingsegmentation, merging, and front addition in connection with a largefile only by using inode information and manipulating a small number ofdata blocks.

In accordance with an aspect of the present invention, there is provideda method for the distributed processing of a file, including selectingat least one first data block to remain without change after distributedprocessing of a file; selecting at least one second data block to bemodified after the distributed processing of a file; pointing, by aninode after the distributed processing of a file, to the first datablock so that the first data block is shared before and after thedistributed processing of a file; and pointing, by the inode after thedistributed processing of a file, to at least one third data blockmodified from the second data block.

The distributed processing of a file may correspond to any one of filesegmentation, file merging, and addition into a front of a file.

If the distributed processing of a file corresponds to the filesegmentation, the third data block may correspond to two data blockscopied from two block fragments of the second data block that has beeninternally fragmented after the file segmentation.

If the distributed processing of a file corresponds to the file mergingand the sum of sizes of data fragments of the internally fragmentedsecond data block is equal to or smaller than a size of one data block,the at least third data block is one in number.

If the distributed processing of a file corresponds to the addition intothe front of a file, the second data block is a first data block of afile before the distributed processing of a file and there is an emptyspace corresponding to a size of new data to be added to the first datablock, the third data block may correspond to a data block that isobtained by moving existing data of the first data block backward andcopying the new data in front of the existing data.

If the distributed processing of a file corresponds to the addition intothe front of a file, the second data block is a first data block of afile before the distributed processing of a file and there is no emptyspace corresponding to a size of new data to be added to the first datablock, the third data block may become an index block, and an inode ofthe index block may sequentially point to a data block including the newdata and the first data block.

In accordance with an aspect of the present invention, there is providedan apparatus for the distributed processing of a file, including ashared data block selection unit configured to select at least one firstdata block to remain without change after distributed processing of afile; a modified data block selection unit configured to select at leastone second data block to be modified after the distributed processing ofa file; a first file distributed-processing unit configured to allow aninode after the distributed processing of a file to point to the firstdata blocks so that the first data block is shared before and after thedistributed processing of a file; and a second filedistributed-processing unit configured to allow the inode after thedistributed processing of a file to point to at least one third datablock modified from the second data block.

The distributed processing of a file may correspond to any one of filesegmentation, file merging, and addition into the front of a file.

If the distributed processing of a file corresponds to the filesegmentation, the third data block may correspond to two data blockscopied from two block fragments of the second data block that has beeninternally fragmented after the file segmentation.

If the distributed processing of a file is the file merging and a sum ofsizes of data fragments of the internally fragmented second data blockis equal to or smaller than a size of one data block, the at least thirddata block is one in number.

If the distributed processing of a file corresponds to the addition intothe front of a file, the second data block is a first data block of afile before the distributed processing of a file and there is an emptyspace corresponding to a size of new data to be added to the first datablock, the third data block may correspond to a data block that isobtained by moving existing data of the first data block backward andcopying the new data in front of the existing data.

If the distributed processing of a file corresponds to the addition intothe front of a file, the second data block is a first data block of afile before the distributed processing of a file and there is no emptyspace corresponding to a size of new data to be added to the first datablock, the third data block may become an index block, and an inode ofthe index block may sequentially point to a data block including the newdata and the first data block.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features and advantages of the presentinvention will be more clearly understood from the following detaileddescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 is a flowchart illustrating a method for the distributedprocessing of a file according to an embodiment of the presentinvention;

FIG. 2 is a diagram illustrating the configuration of the data blocks ofan original file according to an embodiment of the present invention;

FIG. 3 is a diagram illustrating the configuration of data blocks whenan original file having the configuration of FIG. 2 is segmented intotwo files;

FIG. 4 is a diagram illustrating the configuration of the data blocks oftwo original files to be merged according to an embodiment of thepresent invention;

FIGS. 5 and 6 are diagrams illustrating the configurations of datablocks when original files having the construction of FIG. 4 are merged;

FIGS. 7 and 8 are diagrams illustrating the configuration of data blockswhen data is added to the front of an original file having theconstruction of FIG. 4; and

FIG. 9 is a block diagram of an apparatus for the distributed processingof a file according to an embodiment of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Embodiments of the present invention will be described with reference tothe accompanying drawings in order to describe the present invention indetail so that those having ordinary knowledge in the technical field towhich the present pertains can easily practice the present invention. Itshould be noted that same reference numerals are used to designate thesame or similar elements throughout the drawings. In the followingdescription of the present invention, detailed descriptions of knownfunctions and constructions which are deemed to make the gist of thepresent invention obscure will be omitted.

A method and apparatus for the distributed processing of a fileaccording to embodiments of the present invention will be described indetail below with reference to the accompanying drawings.

FIG. 1 is a flowchart illustrating a method for the distributedprocessing of a file according to an embodiment of the presentinvention.

Referring to FIG. 1, in the method for the distributed processing of afile according to this embodiment of the present invention, first, atleast one first data block that will remain without change after thedistributed processing of a file is selected at step S100. Thedistributed processing of a file may correspond to any one of filesegmentation, file merging, and addition into the front of a file.

Thereafter, at least one second data block to be modified after thedistributed processing of a file is selected at step S200.

Thereafter, in order for the first data block to be shared before andafter the distributed processing of a file, an inode after thedistributed processing of a file points to the first data block at stepS300.

Finally, the inode after the distributed processing of a file points toat least one third data block modified from the second data block atstep S400.

If the distributed processing of a file corresponds to filesegmentation, the third data block corresponds to two blocks that arecopied from two block fragments into which the second data block thathas been internally fragmented after the file segmentation. Theconfiguration of data blocks attributable to file segmentation will bedescribed in detail below with reference to FIG. 3.

Furthermore, if the distributed processing of a file corresponds to filemerging and the sum of the data fragments of the internally fragmentedsecond data block is equal to or smaller than the size of a single datablock, the at least one third data block may be one in number. Theconfiguration of data blocks attributable to file merging will bedescribed in detail below with reference to FIGS. 5 and 6.

Furthermore, if the distributed processing of a file corresponds toaddition into the front of a file, the second data block is the firstdata block of a file before the distributed processing of a file, andthe configuration of the third data block may vary depending on whetheror not there is an empty space corresponding to the size of new data tobe added to the first data block.

For example, the second data block is the first data block of a filebefore the distributed processing of a file, and the third data blockcorresponds to a data block that is obtained by moving the existing dataof the first data block backward and then copying the new data in frontof the existing data if there is an empty space corresponding to thesize of new data to be added to the first data block. The configurationof data blocks attributable to addition into the front of a file whenthere is an empty space will be described in detail below with referenceto FIG. 7.

The second data block is the first data block of a file before thedistributed processing of a file, the third data blocks is an indexblock if there is no empty space corresponding to the size of new datato be added to the first data block, and the inode of the index blocksequentially points a data block including the new data and the firstdata block. The configuration of data blocks attributable to additioninto the front of a file if there is no empty space will be described indetail below with reference to FIG. 8.

FIG. 2 is a diagram illustrating the configuration of the data blocks ofan original file according to an embodiment of the present invention.

Referring to FIG. 2, a file 1 ‘f1’ is an original file, and includes aplurality of data blocks, that is, data blocks 1, 2, 3, 4 and 5, and aninode indicative of the file 1 ‘f1’. The inode points to the pluralityof data blocks 1, 2, 3, 4 and 5. In FIG. 3, which will be describedlater, an example in which the file 1 ‘f1’ is segmented into two fileswill be described.

An inode is a data structure that is used in existing Unix-series filesystems. The inode includes information about a file system, such as anormal file or a directory. Each file has a single inode, and the inodeincludes information about a corresponding file, such as an owner group,access inode (e.g., a read, write, or execution right), file type, andan inode number (or an i-number). The files of a file system may beidentified by unique inode numbers. In general, when a file system isgenerated, about one percent of the overall space is assigned to inodes.Since the space for inodes is limited, the maximum number of files of afile system is also limited. In most cases, however, a user feels thatan almost infinite number of files may be generated and managed.

FIG. 3 is a diagram illustrating the configuration of data blocks whenan original file having the configuration of FIG. 2 is segmented intotwo files.

Referring to FIG. 3, the original file having the configuration of FIG.2 is segmented into two files: a file 2 ‘f2’ and a file 3 ‘f3’ havingthe same size. The file 2 ‘f2’ and the file 3 ‘f3’ generate respectiveinodes indicative of the two files, and the generated inodes are setsuch that they point to the data blocks of the file 1 ‘f1’. In thiscase, since the file 1 ‘f1’ is segmented into two files in the middle ofthe data block 3 of the file 1 ‘f1’, two new blocks, that is, a datablock 3′ and a data block 3″, are generated, the front half of the datablock 3 is copied to the data block 3′, and the rear half thereof iscopied to the data block 3″. That is, the data block 3 of the file 1‘f1’ corresponds to the second data block, the data block 3′ and thedata block 3″ generated from the data block 3 correspond to the thirddata block, and the remaining data blocks 1, 2, 4, and 5 correspond tothe first data block. Thereafter, the inode of the file 2 ‘f2’ points tothe data blocks 1, 2 and 3′ of the file 1 ‘f1’, and the inode of thefile 3 ‘f1’ points to the data block 3″, 4 and 5 of the file 1 ‘f1’,thereby completing file segmentation. Accordingly, the file 2 ‘f2’ andthe file 3 ‘f3’ share the data blocks 1, 2, 4 and 5 of the file 1 ‘f1’.If any one of the shared blocks is modified, all the blocks are copiedand then the content of each file is stored.

FIG. 4 is a diagram illustrating the configuration of the data blocks oftwo original files to be merged according to an embodiment of thepresent invention.

Referring to FIG. 4, a file system according to an embodiment thepresent invention merges files by permitting internal fragmentation to afirst block. Accordingly, a file 2 ‘f2’ includes a plurality of datablocks, that is, data blocks 1, 2 and 3, and an inode indicative of thefile 2 ‘f2’. Furthermore, a file 3 ‘f3’ includes a plurality of datablocks, that is, data blocks 4, 5 and 6, and an inode indicative of thefile 3 ‘f3’. A file 4 ‘f4’ that points to all the data blocks of thefiles 2 ‘f2’ and 3 ‘f3’ configured as described above is generated, andthe file 2 ‘f2’ and the file 3 ‘f3’ are merged using the file 4 ‘f4’ intwo forms, as illustrated in FIGS. 5 and 6.

FIGS. 5 and 6 are diagrams illustrating the configurations of datablocks when original files having the construction of FIG. 4 are merged.

Referring to FIG. 5, a file 2 ‘f2’ includes data blocks 1, 2, and 3 andan inode indicative of the file 2 ‘f2’, and the file 3 ‘f3’ includesdata blocks 4, 5, and 6 and an inode indicative of the file 3 ‘f3’.Furthermore, a file 4 ‘f4’ including an inode that points to the datablocks of the file 2 ‘f2’ and the file 3 ‘f3’ is generated. In thiscase, if the sum of the data fragments of the data block 3 of the file 2‘f2’ and the data block 4 of the file 3 ‘f3’ is equal to or smaller thanthe size of one data block, a new data block, that is, one data block 7,is generated. In this case, if a memory location is assigned to a file,the data block 3 of the file 2 ‘f2’ and the data block 4 of the file 3‘f3’ are internally fragmented data blocks that have been assigned andconsumed to maintain block units because all basic input and outputoperations are performed on a block basis. Furthermore, the data block 7corresponds to a third data block, the data blocks 1 and 2 of the file 2‘f2’ and the data blocks 5 and 6 of the file 3 ‘f3’ correspond to afirst data block, and the data block 3 of the file 2 ‘f2’ and the datablock 4 the file 3 ‘f3’ correspond to a second data block. Thereafter,the inode of the file 4 is allowed to point to the data blocks 1, 2, 5,6 and 7 of the file 2 ‘f2’ and the file 3 ‘f3’, thereby completing filemerging.

Referring to FIG. 6, a file 2 ‘f2’ includes data blocks 1, 2, and 3 andan inode indicative of the file 2 ‘f2’, and a file 3 ‘f3’ includes datablocks 4, 5, and 6 and an inode indicative of the file 3 ‘f3’.Furthermore, a file 4 ‘f4’ including an inode that points to the datablocks of the file 2 ‘f2’ and the file 3 ‘f3’ is generated. In thiscase, if the sum of the data fragments of the data block 3 of the file 2‘f2’ and the data block 4 of the file 3 ‘f3’ is larger than the size ofone data block, the file 4 ‘f4’ points to the data block 3 of the file 2‘f2’, the data block 4 of the file 3 ‘f3’ and other data blocks, therebycompleting file merging. In this case, a third data block is notgenerated, the data blocks 1 and 2 of the file 2 ‘f2’ and the datablocks 5 and 6 of the file 3 ‘f3’ correspond to a first data block, andthe data block 3 of the file 2 ‘f2’ and the data block 4 of the file 3‘f3’ correspond to a second data block.

FIGS. 7 and 8 are diagrams illustrating the configuration of data blockswhen data is added to the front of an original file having theconstruction of FIG. 4.

Referring to FIG. 7, front addition may be implemented using an emptyspace because the data block 1 of the file 3 ‘f3’ of FIG. 4 includes theempty space. That is, if there is an empty space corresponding to thesize of new data to be added to the data block 1, the existing data ofthe data block 1 is moved backward and the new data is then copied,thereby completing addition into the front of a file. In this case, thedata blocks 5 and 6 of the file 3 ‘f3’ corresponds to a first datablock, and the data block 4 corresponds to a second data block. In thiscase, the front-added data block portion of the data block 4 correspondsto a third data block.

Referring to FIG. 8, front addition may be implemented by changing thedata block 1 into an index block, that is, a data block 0, because thereis an empty space in the data block 1 of the file 2 ‘f2’ of FIG. 4. Thatis, if there is no empty space corresponding to the size of new data tobe added to the data block 1, the data block 1 is changed into an indexblock and the inode of the index block is allowed to sequentially pointto a data block including the new data and the data block 1, therebycompleting addition into the front of a file. In this case, the datablocks 2 and 3 of the file 2 ‘f2’ correspond to a first data block, thedata block 1 correspond to a second data block, and the data block 0correspond to a third data block.

FIG. 9 is a block diagram of an apparatus 100 for the distributedprocessing of a file according to an embodiment of the presentinvention.

Referring to FIG. 9, the apparatus 100 for the distributed processing ofa file according to the present invention includes a shared data blockselection unit 110, a modified data block selection unit 120, a firstfile distributed-processing unit 130, and a second filedistributed-processing unit 140.

The shared data block selection unit 110 selects a first data block thatwill remain without change after the distributed processing of a file.The distributed processing of a file may correspond to any one of filesegmentation, file merging, and addition into the front of a file.

The modified data block selection unit 120 selects at least one seconddata block to be modified after the distributed processing of a file.

The first file distributed-processing unit 130 allows an inode after thedistributed processing of a file to point to the first data block sothat the first data block is shared before and after the distributedprocessing of a file.

The second file distributed-processing unit 140 allows an inode afterthe distributed processing of a file to point to one or more third datablocks modified from the second data blocks.

If the distributed processing of a file corresponds to filesegmentation, the third data block corresponds to two blocks copied fromthe two block fragments of the second data block that has beeninternally fragmented after the file segmentation.

Furthermore, if the distributed processing of a file corresponds to filemerging and the sum of the data fragments of the internally fragmentedsecond data block is equal to or smaller than the size of one datablock, the number of third data blocks may be one.

Furthermore, if the distributed processing of a file corresponds toaddition into the front of a file, the second data block corresponds tothe first data block of a file before the distributed processing of afile, and the third data block may have a different configurationdepending on whether or not there is an empty space corresponding to thesize of new data to be added to the first data block. For example, ifthe second data block is the first data block of a file before thedistributed processing of a file and there is an empty spacecorresponding to the size of new data to be added to the first datablock, the third data block may correspond to a data block that has beenobtained by moving the existing data of the first data block backwardand then copying the new data in front of the existing data. If thesecond data block is the first data block of a file before thedistributed processing of a file and there is no empty spacecorresponding to the size of new data to be added to the first datablock, the third data block may become an index block and the inode ofthe index block may be allowed to sequentially point to a data blockincluding the new data and the first data block.

As described above, in accordance with the method and apparatus for thedistributed processing of a file according to the present invention,segmentation, merging, and front addition are performed on a large fileby using inode information and manipulating a small number of datablocks, thereby reducing the time it takes to read and write data blocksand the number of times that data blocks are read and written and alsoimproving the efficiency of a file system.

Furthermore, in accordance with the method and apparatus for thedistributed processing of a file according to the present invention,parallelism is increased by segmenting a large file into a large numberof files if there are many available resources, and parallelism isreduced by segmenting a large file into a small number of files if thereare small available resources, thereby making the best use ofparallelism of data blocks and thus improving performance of a filesystem.

Furthermore, in accordance with the method and apparatus for thedistributed processing of a file according to the present invention, afile memory space can be efficiently used by allowing an original fileand a segmented or merged file to share most of data blocks.

Accordingly, in accordance with the method and apparatus for thedistributed processing of a file according to the present invention, ina genome analysis application, the time it takes to perform filesegmentation, file merging and front addition, that is, additional tasksfor data analysis, can be reduced, thereby preventing a main dataanalysis task from being interrupted, and the numbers of segmented filesand merged files can be expected, thereby reducing the burden of a filestorage space.

Although the preferred embodiments of the present invention have beendisclosed for illustrative purposes, those skilled in the art willappreciate that various modifications, additions and substitutions arepossible, without departing from the scope and spirit of the inventionas disclosed in the accompanying claims.

What is claimed is:
 1. A method for distributed processing of a file,comprising: selecting at least one first data block to remain withoutchange after distributed processing of a file; selecting at least onesecond data block to be modified after the distributed processing of afile; pointing, by an inode after the distributed processing of a file,to the first data block so that the first data block is shared beforeand after the distributed processing of a file; and pointing, by theinode after the distributed processing of a file, to at least one thirddata block modified from the second data block.
 2. The method of claim1, wherein the distributed processing of a file corresponds to any oneof file segmentation, file merging, and addition into a front of a file.3. The method of claim 2, wherein if the distributed processing of afile corresponds to the file segmentation, the third data blockcorresponds to two data blocks copied from two block fragments of thesecond data block that has been internally fragmented after the filesegmentation.
 4. The method of claim 2, wherein if the distributedprocessing of a file corresponds to the file merging and a sum of sizesof data fragments of the internally fragmented second data block isequal to or smaller than a size of one data block, the at least thirddata block is one in number.
 5. The method of claim 2, wherein if thedistributed processing of a file corresponds to the addition into thefront of a file, the second data block is a first data block of a filebefore the distributed processing of a file and there is an empty spacecorresponding to a size of new data to be added to the first data block,the third data block corresponds to a data block that is obtained bymoving existing data of the first data block backward and copying thenew data in front of the existing data.
 6. The method of claim 2,wherein if the distributed processing of a file corresponds to theaddition into the front of a file, the second data block is a first datablock of a file before the distributed processing of a file and there isno empty space corresponding to a size of new data to be added to thefirst data block, the third data block becomes an index block, and aninode of the index block sequentially points to a data block includingthe new data and the first data block.
 7. An apparatus for distributedprocessing of a file, comprising: a shared data block selection unitconfigured to select at least one first data block to remain withoutchange after distributed processing of a file; a modified data blockselection unit configured to select at least one second data block to bemodified after the distributed processing of a file; a first filedistributed-processing unit configured to allow an inode after thedistributed processing of a file to point to the first data blocks sothat the first data block is shared before and after the distributedprocessing of a file; and a second file distributed-processing unitconfigured to allow the inode after the distributed processing of a fileto point to at least one third data block modified from the second datablock.
 8. The apparatus of claim 7, wherein the distributed processingof a file corresponds to any one of file segmentation, file merging, andaddition into a front of a file.
 9. The apparatus of claim 8, wherein ifthe distributed processing of a file corresponds to the filesegmentation, the third data block corresponds to two data blocks copiedfrom two block fragments of the second data block that has beeninternally fragmented after the file segmentation.
 10. The apparatus ofclaim 8, wherein if the distributed processing of a file is the filemerging and a sum of sizes of data fragments of the internallyfragmented second data block is equal to or smaller than a size of onedata block, the at least third data block is one in number.
 11. Theapparatus of claim 8, wherein if the distributed processing of a filecorresponds to the addition into the front of a file, the second datablock is a first data block of a file before the distributed processingof a file and there is an empty space corresponding to a size of newdata to be added to the first data block, the third data blockcorresponds to a data block that is obtained by moving existing data ofthe first data block backward and copying the new data in front of theexisting data.
 12. The apparatus of claim 8, wherein if the distributedprocessing of a file corresponds to the addition into the front of afile, the second data block is a first data block of a file before thedistributed processing of a file and there is no empty spacecorresponding to a size of new data to be added to the first data block,the third data block becomes an index block, and an inode of the indexblock sequentially points to a data block including the new data and thefirst data block.